feat(scorers): add citation correctness scorer for RAG outputs - #27

Review feedback on wandb#27, items 1 and 4.

A bracketed token is not the same thing as a citation. Any unresolved bracket
became a fabricated citation, and because that check only ran once something
else had resolved, a correctly cited response failed for containing an array
index, a placeholder, or a markdown link. Citing correctly is what unlocked the
false accusation.

Markers are now classified by shape against the labels the context actually
uses. `_marker_signature` records whether a marker is numeric, hyphenated or
dotted, and a marker only becomes a fabrication candidate when that signature
matches one of the parsed labels. The comparison is derived from the context
rather than hard-coded, so a context whose sources are labelled [TODO] would
make [TODO] a legitimate citation. Markdown-link text never qualifies: it names
a link target, and confirming a URL supports a claim is a different problem.
Link text can still resolve normally when it happens to name a real source.

Resolution now returns a fourth bucket, ignored, for tokens that do not look
like source ids at all. Those are reported in details.ignored_markers so the
shape test is auditable rather than a silent filter.

The all-unresolved branch no longer discards what it computed. Classified
markers are carried into details on every path. When the context parsed into
labelled blocks and a marker matches how those blocks are named, that is a
fabricated citation and now fails with score 0 rather than skipping the row;
the judge is not called, so raw_score is None. Structurally mismatched markers,
such as numeric markers against slug labels, stay un-assessed as a likely
citation-format mismatch.

Adds regression tests for the four reported cases (arr[0], [x], [TODO],
[Wikipedia](url)), a control that a genuine fabrication alongside stray brackets
still floors, link text resolving when it names a real source, the new
fail-not-skip path, structural mismatch staying un-assessed, and unlabelled
context being unable to produce a fabrication.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review feedback on wandb#27, items 2 and 3. Both are the same idea from opposite
ends: the judge should only ever deal with citations the response made.

Evidence haystacks were keyed on every parsed source block rather than the
resolved ones, so the judge could return a verdict about a block the response
never cited, have its span verify against that block's text, and be accepted.
A response citing only [adverse-action] scored 1.0 on evidence tagged
fair-lending. Haystacks are now built from the resolved markers alone; the
existing discard path in _verified_citation_spans then drops anything else and
counts it in discarded_evidence_spans, so no new logic was needed. Misattributed
evidence still verifies against the whole context, since by definition it comes
from a different block; only which markers may appear at all has narrowed.

Discarding bad verdicts afterwards is not sufficient on its own. The judge
returns one holistic score, so a marker it graded and disliked drags that score
down even when its span is thrown away. The prompt now names the resolved
markers and instructs the judge to grade only those, and the template no longer
says "for every marker", which contradicted the scope it is given.

Adds tests that evidence for an uncited block is discarded for both supported
and misattributed items, that evidence for a cited block is still accepted, that
the prompt scopes grading to resolved markers while excluding ambiguous ones and
stray brackets, and that the scope and fabricated blocks can both be appended
without breaking the JSON example's escaped braces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… fixes

Two claims no longer held. The docstring listed array indexing as a live
limitation, when a marker that does not resemble the context's labels is now
reported under ignored_markers and has no effect on the score. It also said any
row whose citations cannot be resolved is returned un-assessed, which stopped
being true when label-shaped markers that name nothing began failing outright.

Replaces both with what the class now does: how markers are compared against the
shape of the labels the context uses, the three outcomes for a marker that names
no block, and the narrower set of conditions under which a row is left
un-assessed. Records that the comparison is only as sharp as the labels are
distinctive, since against sources named 1, 2, 3 an array index really is
indistinguishable from a citation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review round 2 on wandb#27, items 1, 2, 5 and 6 of that pass.

The marker-shape check was too coarse for label styles this class claims to
support. With sources labelled [doc], both [x] and [TODO] share the signature of
the real label; with sources labelled [1], so does the 0 in arr[0]. Each was
classified as fabricated, and the deterministic floor then overrode a judge
verdict of 3/pass into 0/fail. The existing regressions only used hyphenated
labels, so none of this was caught.

A grammar is now checked before any accusation is made. Source ids only support
the distinction when they carry a separator that incidental brackets do not, so
labels are required to contain "-", "." or "_". Where they do not, no marker can
be a fabrication candidate, and a row with nothing resolved is returned
un-assessed under a new reason, unsupported_label_style, rendered in the coverage
report as "source labels are not distinguishable from ordinary text". This does
lose genuine fabrications under bare-word and bare-number schemes. That is the
intended trade: a missed finding is recoverable, a false compliance failure is
not, and a caller wanting accusation there can supply a stricter citation
pattern.

Classification was also order-dependent. Extraction de-duplicated on the first
occurrence and kept that occurrence's link flag, so "[fake-id](url) ... [fake-id]"
was ignored while the reverse order was called a fabrication. The flag is now
collapsed across every occurrence: a marker written as a link anywhere is treated
as a link throughout. Mixed forms are weak evidence of a citation attempt, so the
conservative reading is the one that declines to accuse.

Empty and duplicate source blocks are no longer treated as valid. A label whose
body is empty cannot support any claim, yet citing it resolved and could pass on
evidence that never verified. A repeated label silently kept the first block
while the judge read both, so evidence quoted from the later block was rejected
and the score passed anyway. Both are now excluded from the parsed blocks; since
their labels still appear in the context, a citation naming one is reported as
ambiguous rather than fabricated, and the judge is not called.

Adds regressions for all three label styles, both orderings of the mixed-form
case, each of the three separators, empty and duplicate blocks, and controls that
structured labels still accuse, that a surviving block stays gradeable alongside
a dropped one, and that unlabelled context keeps the generic reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review round 2 on wandb#27, items 3 and 4 of that pass.

The judge's score went straight into the normaliser with no checks. Because
min(1.0, nan) is 1.0 in Python, a reply of "NaN" clamped upward into a perfect
compliance result, and an out-of-range 99 did the same. A non-numeric score or a
missing key raised out of float() as an unhandled exception, which the review did
not reach but is the same defect. _valid_judge_score now requires a finite number
within the rubric range and returns None otherwise. It runs before the normaliser
rather than inside it, so an unusable score becomes an un-assessed row instead of
a wrong number or a traceback, and so this stays independent of wandb#50, which is
fixing non-finite handling at the ScoreNormalizer boundary.

Coverage was never checked either. A score of 3 across two resolved citations
passed while the judge had returned a verdict for only one of them, so the row
reported a confident result for a citation nobody assessed. Every resolved marker
must now carry a verdict, measured against the verified lists rather than the raw
reply: a verdict whose evidence could not be found in the row is discarded, and a
discarded verdict has established nothing, so counting it would let invented
evidence satisfy the requirement it exists to enforce. A misattributed verdict
counts as covered, since this asks whether every citation got a checkable answer,
not whether the answers were favourable.

Invalid or incomplete output is un-assessed under invalid_judge_score or
incomplete_judge_verdicts, reported as "the judge returned an unusable score" and
"the judge did not assess every citation". The two are kept separate because they
have different remedies: one points at the judge model, the other at the prompt.
Where a fabrication has already been established deterministically the row still
fails with score 0 rather than becoming un-assessed, since an unusable reply must
not rescue a response that provably cited a nonexistent source.

Existing fixtures returned an empty verdict list alongside a score of 3, which is
exactly the shape now rejected, so they are rebuilt through a helper that returns
a verifiable verdict for every resolved marker. Three tests that fed only
unverifiable evidence now assert the stronger outcome: bad evidence cannot
satisfy coverage, so the row is un-assessed rather than scored.

Adds regressions for every malformed score, missing and partial verdicts, a
misattributed verdict counting as coverage, the deterministic override, and both
new coverage-report reasons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…loor explain itself

Review round 2 on wandb#27, items 5 and 6 of that pass.

A response citing both a real block label and a marker that only appears inside
that block was graded on the first and reported at 1.0, passed and assessed,
while the second sat unjudged in ambiguous_citations. Coverage validation did not
catch it because ambiguous markers never reach the judge, so nothing required a
verdict for them. The row was partly assessed and reported as a complete one.

An ambiguous marker now makes the row un-assessed under partial_citation_coverage,
reported as "only some of the response's citations could be resolved". The binary
assessed field has no partial state, and the alternative of adding one would
change ScorerResult and the aggregation path for every scorer in the toolkit, so
the narrower reading is taken here.

Ignored markers deliberately do not count. Those were determined not to be
citations at all, so treating them as blocking would undo the stray-bracket fix
and send any response containing an array index back to un-assessed. A fabrication
still fails rather than being downgraded into a coverage gap, since a proven
finding outranks an ungradeable one.

The deterministic floor also contradicted its own explanation. The judge is not
told the outcome, so a row could return score 0 and passed=False carrying the
judge's text that the response was fully supported. The floor now leads with the
deterministic finding and names the absent sources, with the judge's own words
following rather than dropped, and the unmodified judge text preserved in
details.judge_explanation for audit.

Adds regressions for the partly resolved row, the ignored-marker control, a
fabrication outranking ambiguity, the floor explanation naming the fabricated
sources while retaining the judge text, unfloored rows keeping the judge
explanation unchanged, and the new coverage-report reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…tyle

Review round 3 on wandb#27, findings 2 and 3.

A source label was matched without checking what followed the closing bracket, so
Markdown at the start of a line parsed as a source. "[docs](https://example.com)"
produced a docs block whose body was the URL, and a response citing [docs] was
graded against it for a full pass. Reference definitions ("[docs]: https://...")
behaved the same way. The label pattern now requires whitespace or end of line
after "]", which excludes "](" as asked and also covers "]:" and any other
non-boundary character. The character class is unchanged, so the grammar shared
with wandb#37 still matches; only the boundary tightens.

The shape signature and the separator list had drifted apart. The separator list
counted "_" as structure while the signature recorded only digits, hyphens and
dots, so [source_one] and [TODO] shared a signature and an ordinary editorial
token was floored to 0. The signature now derives a bit from every separator, so
the two cannot disagree again.

Distinguishability was also decided for the context as a whole, which let one
unstructured label disable accusation for every other style present: a context
labelled [doc-1] and [2] passed a missing [doc-99]. _accusable_signatures
replaces the global check and reports the shapes that carry structure, so each
style is judged on its own terms.

Shape is now checked before inline presence, so bracket notation copied out of
the context is recognised as not being a citation instead of becoming ambiguous
and blocking the row as partial coverage. The check is conditional on a
structured style being present: with no such style there is nothing to judge
shape by, and a marker that does appear bracketed in the context is still a
plausible citation, so the ambiguous tier survives for duplicate and empty
labels.

Adds regressions for Markdown links, reference definitions, a missing boundary, a
real block following Markdown, underscore labels in both directions, mixed label
styles in both directions, bracket notation copied from source text, and a
control that a label-shaped marker inside a passage is still ambiguous.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…JSON-safe

Review round 3 on wandb#27, finding 5.

JSON booleans passed the numeric check. bool subclasses int, so float(True) is
1.0 and a reply of true was accepted as a rubric score of 1. Booleans are now
rejected before the numeric conversion, since any test based on float() lets them
through.

A reply with a non-object top level raised instead of producing a controlled
result. _call_judge returns whatever the response parsed to, and valid JSON is
null or [] often enough that reading a score off one crashed with AttributeError.
Non-object replies are now returned un-assessed with the reply recorded, in the
same form as any other unusable output.

Rejected raw values are kept for audit but are no longer stored verbatim. A float
NaN or infinity in details["raw_score"] leaks non-standard JSON into assessment
reports, which is the contract knisar asked for on wandb#50: the value is stored as a
string under rejected_raw_score, and raw_score is None because no valid score
exists. Adds a test that every un-assessed path serializes under
json.dumps(allow_nan=False), not only the deterministic one.

The skip reason is renamed from invalid_judge_score to invalid_judge_output. It
now covers replies that are not objects at all, which are not score problems, and
the coverage report reads "the judge returned unusable output".

Also fixes the test helper, which collapsed falsy judge replies to an empty dict
through "result or {}" and so could never deliver the None and [] cases this
finding is about.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…rence

Review round 3 on wandb#27, findings 1 and 4. This changes the unit of judgment
rather than adding another guard, and it changes the judge contract.

Citations were collapsed by marker, so two claims citing one source were a
single thing to verify and one verdict carried both: an unsupported claim passed
on its neighbour's evidence. The unit is now the occurrence. Extraction returns
every occurrence in order, each carrying the claim it is attached to, and every
occurrence needs its own verified verdict.

The claim is derived from the marker's position rather than supplied by the
judge. When the judge chose the response span, verification could only confirm
the text appeared somewhere in the response, so swapping the spans of two
citations still verified and both wrong associations passed. The judge is now
given a numbered list of citations with their claims and answers only the
question it is qualified for, whether the named block supports that claim.
Mis-association is no longer expressible rather than merely detectable.

Verdict coherence is enforced. Exactly one verified outcome is required per
occurrence, and a second outcome for the same occurrence is a contradiction
rather than something to resolve by taking whichever verified. The judge's score
must sit in the band its own verified verdicts imply, since a verified
misattribution alongside a score of 3 previously returned a full pass. Validating
the score was chosen over deriving it so the rubric keeps its band 2, which
outcomes alone cannot express.

Misattribution now means what it says. The judge must name the block that does
support the claim, that block must differ from the one cited, and the evidence
must come from it. Evidence drawn from the cited block proves the claim is
supported by what was cited, the opposite of misattribution, and was accepted
before because spans were verified against the whole context.

Two further un-assessed reasons, contradictory_judge_verdicts and
judge_score_contradicts_verdicts, are classified for the coverage report.

Existing fixtures spoke the old two-list contract and are migrated. Several now
assert stronger outcomes: a single verdict no longer covers two occurrences, and
misattribution evidence from the cited block leaves the citation unestablished.

Adds regressions for both reported reproductions, contradictory verdicts, a score
disagreeing with its verdicts, misattribution evidence from the cited and from an
unknown block, the numbered prompt, and controls that two claims citing one
source pass when each is verdicted and that claims stay bound to their citation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…im spans

Follow-up within round 3 finding 1, replacing the claim derivation added in the
previous commit.

Slicing the text between markers to recover each claim was fragile on ordinary
prose. A mid-sentence citation truncated to a fragment, "The rule [x] requires
notices" grading only "The rule"; a citation opening a sentence produced an empty
claim; two citations in one sentence produced two fragments; and a citation
following another sentence swept that sentence in. The judge would have been
grading text the response never asserted, which is a worse failure than the
mis-association it was meant to prevent.

The response shown to the judge now carries an occurrence tag after each marker,
so the judge reads the prose as written and answers per number. Deciding where a
claim begins and ends is left to the reader that can actually do it, while the
binding between a verdict and a citation stays exact because the number is
attached to the marker in the text. The tag characters cannot be matched by the
citation pattern, which requires an alphanumeric immediately after "[".

claim_span becomes advisory. The judge quotes the claim it graded, and the quote
is recorded when it appears verbatim in the response because a report reads
better naming the claim than an index. It never binds the verdict, so a swapped
or invented quote costs at most a blank line in the report: a swap cannot move a
verdict between citations, and an invented quote is discarded while the verdict
stands on its verified context evidence.

Adds regressions that a swapped claim quote cannot move a verdict, that an
invented one is discarded while the verdict survives, and that the response
reaches the judge with each occurrence tagged in place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Not reported on the pull request. Found by probing the round-3 work the way the
review has been probing it, before pushing.

A response that already contained the occurrence tag characters was annotated on
top of them, so the prompt carried two identical tags and a verdict could bind to
a sequence the model wrote itself rather than to a citation. Existing tag
characters are now stripped before annotating, leaving exactly one per citation.

An occurrence number of true or 1.0 was accepted. bool subclasses int and
1.0 == 1, so either indexed occurrence 1 and a verdict for a citation that was
never named could be credited to the first one. This is the same shape as the
boolean score accepted by float(), so the index now has to be a real int.

A single-character evidence span verified. "e" occurs in almost any block, so a
verdict could satisfy verification while corroborating nothing. Spans shorter
than two characters are rejected, which leaves short but real evidence such as
"8%" intact.

The sweep also covered occurrence numbers out of range, verdict containers that
are not lists, bracketed and mis-cased supporting markers, score band edges,
mixed outcomes across occurrences, identical and nested source blocks, claim
quotes that include the occurrence tag, and a response consisting only of a
citation. Those all behaved correctly and are left as they are.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Found by auditing every judge-supplied field rather than only the three the
sweep had already turned up.

str() on the raw value coerced anything into the report, so a reply of null read
as the explanation "None", a dict arrived as "{'a': 1}", and a number as its
digits. The explanation is user-facing text in a compliance report, so a Python
repr reaching it is the same defect as the boolean score and the boolean
occurrence index: a judge-supplied value used without a type check.

Only a string is carried through now; anything else leaves the explanation empty
rather than filled with a representation of the reply.

The audit covered every field read from the judge: score, verdicts, occurrence,
outcome, context_span, supporting_marker, claim_span and explanation. The other
seven already guard their types.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Self-review of the local commits before pushing.

The partial-coverage guard ran after the judge call, so a row containing an
ambiguous marker was billed for a verdict that was then discarded. Every other
guard in this scorer short-circuits first, and this one was added later without
following that shape. Moved ahead of the call, keeping the exemption for a
proven fabrication, which still outranks a coverage gap and must reach the floor.

The review also checked occurrence numbering when an earlier citation is
unresolved, discarded-span arithmetic against contradictory and unknown
verdicts, markdown-link citations that resolve, a marker resolving under two
different cases, and the removal of the helpers the occurrence model replaced.
Those behaved correctly and are unchanged.

Adds a regression that an ambiguous marker leaves the judge uncalled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Occurrence numbers were the ordinal among every bracketed token, including ones
this scorer had already decided were not citations. A response opening with
arr[0] therefore presented its only real citation to the judge as occurrence 2,
with no occurrence 1 anywhere in the prompt, and a response containing several
stray brackets produced a scope list with gaps in it.

The numbering was internally consistent, since the scope list and the inline
tags used the same values, so nothing was mis-graded. It just asked the judge to
read a sequence that skipped numbers for reasons it could not see.

Resolved occurrences are renumbered contiguously at resolution. Every consumer of
the index reads from the resolved list, so there is no second numbering space to
keep in step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ion intent

Review round 4 on wandb#27, findings 1, 2 and the classification case raised
alongside them.

The markdown-link flag was collapsed across every occurrence of a marker, so one
link occurrence marked them all as link text and a bare fabrication elsewhere in
the response was never accused: a valid [doc-1], a bare [doc-99] and a linked
[doc-99] together returned a full pass. The flag is now kept per occurrence. The
collapse had been guarding an order dependence that existed only while extraction
de-duplicated markers, which the occurrence model removed, so it had been doing
nothing but harm since. Order independence still holds, because the occurrences
are distinct either way.

Labels excluded from parsing were then invisible. An empty or repeated
line-leading label is not citable, but it is still a label the context declares,
and a citation naming one fell through to the shape filter and was classed as not
a citation at all. A row citing a dropped [dup] alongside a valid block therefore
passed. Rejected labels are now preserved separately and a citation to one is
ambiguous, checked before any shape filtering, so the row is reported as partly
assessed.

The separator signature established syntactic similarity, not citation intent.
[0-1], [x-y] and [1.2] are indistinguishable from [doc-1] under it, so ordinary
notation was accused and floored an otherwise valid pass. A marker is now
accusation-grade only if some part of it reads as a word, meaning two or more
letters, which separates a source id from an interval or a coordinate pair.
Source-shaped tokens inside fenced or inline code are excluded from extraction
entirely; code spans are located rather than stripped, so marker offsets stay
valid for annotation. A genuinely short id such as a-1 stops being accusable,
which errs towards not accusing.

Verified with a 240-case cross-product of label style, stray token, fabrication
and judge-reply validity rather than per-feature cases, since every finding in
this round was a composition of two or more features. Three invariants hold with
no violations: an accusable label style never passes a fabrication, a stray token
never fails an otherwise clean row, and the clean baseline always passes. The
fourth, that a malformed reply cannot rescue a proven fabrication, still fails
and is finding 8, addressed next.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review round 4 on wandb#27, findings 3 and 8. Both broke the same promise: a
fabrication established in Python produces a deterministic failure, and nothing
the judge does can undo it.

The prompt and the coherence rule contradicted each other. The fabricated block
told the judge to factor absent sources into its score, while the coherence rule
expected a score reflecting only the resolved verdicts. A judge that obeyed the
prompt and scored 0 alongside a supported verdict was therefore recorded as
having produced faulty output, and a judge that ignored the instruction and
scored 3 was treated as coherent. The instruction is removed: the judge is told
which markers are fabricated so it does not try to verify them, and scores only
the citations it was asked to grade. The deterministic floor owns the fabricated
result, which is already the architecture.

A reply that was not a JSON object returned un-assessed even when a fabrication
had already been established, turning a proven failure into a coverage gap. That
guard was added separately and never grew the fabrication branch the other
rejection paths have. Both paths now route through one method, so the rule has a
single owner rather than being restated per call site.

Evidence that did verify is retained on a rejected reply rather than replaced
with empty lists. The reply as a whole being unusable does not make the verdicts
that checked out stop being evidence, and discarding them lost the verified
support the review specifically called out.

Cross-tested over 504 combinations of label style, stray token, fabrication form
and judge-reply validity. Six invariants hold with no violations: an accusable
style never passes a fabrication, a mixed link form fails exactly like a bare
one, a stray token never fails a clean row, a bad reply never rescues a proven
fabrication, the clean baseline always passes, and a dropped label always blocks
assessment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…onour wandb#22

Two review items in _verified_verdicts.

Duplicate evidence across blocks created a false misattribution. Overlapping
retrieved blocks are ordinary - sliding-window chunking, shared boilerplate, two
documents quoting one rule - so the same sentence can sit in both. A misattributed
verdict was accepted whenever its evidence verified against the block named in
supporting_marker, without asking whether the cited block contained that evidence
too. Where both blocks carry it, the evidence cannot distinguish them, and the
cited block visibly supporting the claim is the opposite of misattribution. That
turned a correct citation into an assessed failure, the worst direction for a
compliance report to be wrong in.

The round-3 fix moved this check from the whole context to the supporting block
but never added the converse, so half the rule was missing. A misattributed
verdict is now rejected when its evidence also verifies against the cited block.
The verdict is dropped like any other unverified one, leaving the occurrence
uncovered and the row un-assessed, rather than being rewritten to supported: the
judge did not return that verdict and this scorer does not invent one. Evidence
unique to the supporting block still convicts.

The evidence output also diverged from the shape agreed on wandb#22. Verdicts carried
a `claim` key where the issue specifies `response_span`, the field
_verified_evidence_spans already emits for GroundednessScorer, so a consumer
reading evidence across scorers found a different name here. It is renamed, and
omitted rather than emitted as "" when the judge's quote does not verify, so an
unverifiable quote is not read as a claim that was blank. The verdict still
stands on its context evidence and is still bound by occurrence number, keeping
the advisory treatment settled in round 3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Occurrence tags were written with a fixed delimiter pair, and any of those
characters already in the response were deleted before the tags were inserted.
Deleting them kept what sat between them, so a response reading "the rate is
[1]5%" reached the judge as "the rate is 15%" and the judge graded a number the
model never wrote. Where the context said 15% and the response claimed 5%, the
mutation manufactured agreement and the row passed: a false pass on exactly the
kind of error this scorer exists to catch. It is not only digits - the default
pair is the white square bracket of denotational semantics, so deleting it also
rewrites set and interval notation.

The delimiters are now chosen per response from a candidate list, taking the
first pair absent from it, falling back to private-use codepoints that a finite
response cannot exhaust. Annotation is purely additive: removing the inserted
tags returns the response verbatim, which is the property the tests assert
rather than any particular output. Nothing needs stripping, because nothing can
collide.

Since the pair is no longer fixed, the prompt cannot name one. The sentence
describing it moves out of the template into CITATION_TAG_BLOCK, formatted with
the chosen delimiters and appended alongside the scope and fabricated blocks.
That is the mechanism those blocks already exist for: the base template escapes
the braces of its JSON example and super() collapses them, so a second format
pass over the whole prompt would read them as fields and raise. It also reads
better there, immediately before the block that tells the judge to return one
verdict per number.

The test asserting the old behaviour asserted the bug - that the tag characters
were removed from the response - and is replaced by the round-trip property.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…obey it

The comment explaining why the occurrence tag cannot be read as a citation was
left above the old fixed-delimiter constants and described characters that no
longer exist. The constraint it recorded still binds: a candidate pair using "["
or "]" would make the citation pattern match the tag itself, so annotating a
response would manufacture citations nobody wrote.

Folded into the comment on the candidate list and asserted in a test, so a pair
added later cannot quietly break annotation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A rejected judge reply kept its audit record only when a fabrication happened to
be present. With one, the verdicts that verified, the judge's own score and the
discarded count all reached details; without one the row routed through
_unassessed and carried none of it, leaving a reviewer with a skip reason and
nothing to check it against.

Whether a fabrication exists has nothing to do with why the reply was rejected,
so the record should not depend on it. This was already half of an earlier
review note - that a rejected reply "discards the verified support details" -
which was fixed on the branch that had a fabrication and left on the branch that
did not.

The cost is clearest with two citations where one verdict verifies and the other
does not: the good verdict was thrown away because its neighbour failed, so the
report could not show which citation the judge got right.

Both branches now carry the judge's score, the judge's text, the verdicts that
verified and the number discarded. On an un-assessed row the verdicts go under
verified_verdicts rather than supported_citations, because nothing there was
assessed and a populated results key would read as though it had been.

The judge's explanation is now read before the coherence checks rather than
after, since a rejected reply still has to report what the judge said.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ntains

The verifier rejects a misattributed verdict whose evidence also verifies
against the cited block, because text both blocks contain cannot show which of
them supports the claim. The prompt asked only that the quote come from the
supporting block, so a judge could satisfy the instruction and still be rejected.

That costs a real finding. Where two blocks share boilerplate and disagree on a
figure, a judge can reach the correct verdict and quote the shared sentence to
justify it. The verdict is then dropped and the row goes un-assessed rather than
failing, so a genuine misattribution goes unreported for the judge's choice of
quote rather than for its judgement.

Stating the requirement lets the judge satisfy it: with two blocks differing
only on a rate, the discriminating quote is the rate itself. This addresses the
cause rather than widening what the verifier accepts, which would mean trusting
evidence that does not establish what the verdict claims.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The review note that the fabrication contract contradicts the prompt was only
half addressed. A block was added telling the judge that markers naming no
source are handled outside its score, but the rubric band it contradicts was
left in place: band 0 still read "a citation naming a source absent from the
Context". Both sentences reached the judge in the same prompt.

A judge obeying the rubric scored 0 alongside a supported verdict, which the
coherence rule rejects as judge_score_contradicts_verdicts. The floor still
produced the right final answer, so the row was not wrong, but the reply of a
judge that followed the instructions was thrown out and its verdicts with it.

Band 0 now describes misattribution of the central claim, which is the only
thing left for it to mean once fabrications are scored deterministically, and
the instruction to score only the citations listed is stated in the rubric
itself rather than only in a block appended when a fabrication happens to exist.

Verified against the rest of the prompt: every rubric band, the verbatim
evidence rule, the one-verdict-per-citation rule, the cited-block requirement
and the claim_span-without-its-tag rule now each match what the verifier
enforces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The citation pattern used \s* around the id, and \s matches more than a space:
newline, carriage return, form feed, vertical tab, and the Unicode separators.
A bracketed value wrapped across lines was therefore read as a citation.

That is expensive rather than merely wrong. A marker spanning lines names no
source, and if it resembles the context's label style the fabrication floor
clamps the row to zero, so a response whose only real citation was correct
fails over its formatting. The match also spans the newlines, so the occurrence
tag lands after the closing bracket on a later line and the judge is asked to
grade a number attached to no claim.

A citation marker is written inline. Restricting the surrounding whitespace to
[ \t] matches that, and matches _SOURCE_LABEL_PATTERN, which wandb#37 already fixed
the same way on the context side after hitting the same defect. Both sides now
agree on what counts as one bracketed token, which is the point: they read the
same syntax on either side of the row.

Regressions cover both paths, since the label half is only correct while the
shared pattern stays that way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The exclusion matched a closed triple-backtick fence and a single-backtick
span, and nothing else. Three standard forms fell through: a tilde fence, which
is what a writer reaches for when the code itself contains backticks; a
double-backtick span, where the old pattern matched the empty string between
each pair of ticks and left the content outside both; and an unclosed fence,
which a reply truncated at a token limit leaves behind.

Missing one is expensive rather than merely incomplete. A marker inside code
names no source, so if it resembles the context's label style the fabrication
floor clamps the row to zero: a response that cited correctly and also showed a
code sample is reported as a compliance failure over its formatting.

Fences now open with three or more backticks or tildes and close with a run of
the same character. One opening a line may also run to the end of the response,
since a truncated block is still code. One opened mid-line must close, because
running to the end there would let a stray run in prose - "see ``` for fences" -
hide every citation after it. Inline spans close on a run of the same length as
the opener, so ``x`` is one span rather than two empty ones either side of x.
Fences are resolved first and inline spans sought only outside them.

Regressions cover ten code forms and six shapes of prose that only resemble
code, because over-excluding loses coverage as quietly as under-excluding
manufactures findings.

Indented code blocks are still not recognised. Four-space indentation also
marks list continuations, so excluding it would silently stop grading a real
citation inside a nested list, and model output overwhelmingly fences. Raising
it rather than deciding it here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…haracters

The tag search tried four named delimiter pairs and then the private-use range,
and returned the first two free codepoints. That is 6407 single characters in
total, and a response is free to contain all of them - roughly 19 KB, well
inside any model's output limit. The list came back empty and indexing it raised
IndexError before the judge was called, so the failure left the caller with an
exception rather than a result. Every other failure in this scorer degrades to a
named coverage gap; this one aborted the run.

The comment above that line asserted a response could not exhaust the range. It
was wrong, and the code depended on it being right, which is the same mistake as
the rubric band that contradicted the verifier: reasoning recorded in a comment
instead of enforced in code.

Single characters are now a preference rather than the whole search. Once they
are spent the delimiters are lengthened by repetition, which cannot be
exhausted: a finite text has a longest run of any given character, so one
repetition beyond it appears nowhere. The response is still never altered and
annotation still round-trips.

Lengthening rather than giving up keeps the row graded. Returning un-assessed
whenever a response happened to contain unusual characters would lose real
citation coverage to a parser limit.

The caller handles a None tag with an un-assessed row and a registered skip
reason even though the search makes it unreachable, because the defect being
fixed here came from trusting an invariant rather than checking it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The tag block told the judge that the delimiter characters appear nowhere else
in the response. That was true while a tag was always a single character on
each side. It stopped being true once the tag lengthens by repetition: the
characters then appear throughout the response, which is precisely why the tag
was lengthened, and only the sequence is absent.

The sentence exists so the judge can tell a tag from the model's own text, so
it was false exactly where it does the most work. A judge told a stray bracket
cannot occur, while reading a response full of them, has been given a reason to
treat one as a delimiter.

It now claims what is actually guaranteed - that the exact sequence appears
nowhere else - and says outright that text resembling part of the tag is the
model's own.

Found by re-running the prompt audit against a response that exhausts the
single-character tag search, rather than against the simple fixture where the
old wording happened to hold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…tom labels

Three parsing cases, two of which passed a row that should have failed.

A fence closes on a run of the same character at least as long as the opener,
not exactly as long. The closing rule was a backreference, which can only demand
an equal run, so a block opened with three backticks and closed with four looked
unclosed. The mask then ran to the end of the response and hid every citation
after the block, and an unsupported one among them took a full pass. Because the
rule is a comparison rather than an equality, the fences are scanned line by
line now instead of matched with one expression. A shorter run still does not
close a longer fence.

A code span may cross a line ending. The inline body excluded newlines, so a
marker inside a legitimate multiline span was read as a citation and, resembling
the context's labels, failed a response that had cited correctly.

Resolution collected rejected source labels with the default pattern whatever
the scorer was configured with. A context declaring its sources as
<<source-id>> therefore had nothing rejected, and a citation naming a duplicated
or empty label fell through the shape filter and passed unverified, while the
same shape under the default syntax was correctly reported as ambiguous. Both
patterns now come from the scorer, so the classification holds for an overridden
syntax as well. That guarantee was added for dropped labels in an earlier round
and only ever worked for one configuration.

The first two came from testing the implementation rather than the format: the
fence tests were all symmetric and the inline body was restricted to a single
line on a guess about safety, so the tests inherited the same blind spots. These
follow CommonMark's rules instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>